94 research outputs found
STEER: Unified Style Transfer with Expert Reinforcement
While text style transfer has many applications across natural language
processing, the core premise of transferring from a single source style is
unrealistic in a real-world setting. In this work, we focus on arbitrary style
transfer: rewriting a text from an arbitrary, unknown style to a target style.
We propose STEER: Unified Style Transfer with Expert Reinforcement, a unified
frame-work developed to overcome the challenge of limited parallel data for
style transfer. STEER involves automatically generating a corpus of
style-transfer pairs using a product of experts during decoding. The generated
offline data is then used to pre-train an initial policy before switching to
online, off-policy reinforcement learning for further improvements via
fine-grained reward signals. STEER is unified and can transfer to multiple
target styles from an arbitrary, unknown source style, making it particularly
flexible and efficient.
Experimental results on a challenging dataset with text from a diverse set of
styles demonstrate state-of-the-art results compared to competitive baselines.
Remarkably, STEER outperforms the 175B parameter instruction-tuned GPT-3 on
overall style transfer quality, despite being 226 times smaller in size. We
also show STEER is robust, maintaining its style transfer capabilities on
out-of-domain data, and surpassing nearly all baselines across various styles.
The success of our method highlights the potential of RL algorithms when
augmented with controllable decoding to overcome the challenge of limited data
supervision.Comment: for associated code, see
https://github.com/shallinan1/STEERStyleTransfe
Symbolic Knowledge Distillation: from General Language Models to Commonsense Models
The common practice for training commonsense models has gone
from-human-to-corpus-to-machine: humans author commonsense knowledge graphs in
order to train commonsense models. In this work, we investigate an alternative,
from-machine-to-corpus-to-machine: general language models author these
commonsense knowledge graphs to train commonsense models. Our study leads to a
new framework, Symbolic Knowledge Distillation. As with prior art in Knowledge
Distillation (Hinton et al., 2015), our approach uses larger models to teach
smaller models. A key difference is that we distill knowledge symbolically-as
text-in addition to the neural model. We also distill only one aspect-the
commonsense of a general language model teacher, allowing the student to be a
different type, a commonsense model. Altogether, we show that careful prompt
engineering and a separately trained critic model allow us to selectively
distill high-quality causal commonsense from GPT-3, a general language model.
Empirical results demonstrate that, for the first time, a human-authored
commonsense knowledge graph is surpassed by our automatically distilled variant
in all three criteria: quantity, quality, and diversity. In addition, it
results in a neural commonsense model that surpasses the teacher model's
commonsense capabilities despite its 100x smaller size. We apply this to the
ATOMIC resource, and share our new symbolic knowledge graph and commonsense
models
In Search of the Long-Tail: Systematic Generation of Long-Tail Knowledge via Logical Rule Guided Search
Since large language models have approached human-level performance on many
tasks, it has become increasingly harder for researchers to find tasks that are
still challenging to the models. Failure cases usually come from the long-tail
distribution - data that an oracle language model could assign a probability on
the lower end of its distribution. Current methodology such as prompt
engineering or crowdsourcing are insufficient for creating long-tail examples
because humans are constrained by cognitive bias. We propose a
Logic-Induced-Knowledge-Search (LINK) framework for systematically generating
long-tail knowledge statements. Grounded by a symbolic rule, we search for
long-tail values for each variable of the rule by first prompting a LLM, then
verifying the correctness of the values with a critic, and lastly pushing for
the long-tail distribution with a reranker. With this framework we construct a
dataset, Logic-Induced-Long-Tail (LINT), consisting of 200 symbolic rules and
50K knowledge statements spanning across four domains. Human annotations find
that 84% of the statements in LINT are factually correct. In contrast, ChatGPT
and GPT4 struggle with directly generating long-tail statements under the
guidance of logic rules, each only getting 56% and 78% of their statements
correct. Moreover, their "long-tail" generations in fact fall into the higher
likelihood range, and thus are not really long-tail. Our findings suggest that
LINK is effective for generating data in the long-tail distribution while
enforcing quality. LINT can be useful for systematically evaluating LLMs'
capabilities in the long-tail distribution. We challenge the models with a
simple entailment classification task using samples from LINT. We find that
ChatGPT and GPT4's capability in identifying incorrect knowledge drop by ~3% in
the long-tail distribution compared to head distribution
SODA: Million-scale Dialogue Distillation with Social Commonsense Contextualization
Data scarcity has been a long standing issue in the field of open-domain
social dialogue. To quench this thirst, we present SODA: the first publicly
available, million-scale high-quality social dialogue dataset. By
contextualizing social commonsense knowledge from a knowledge graph, we are
able to distill an exceptionally broad spectrum of social interactions from a
large language model. Human evaluation shows that conversations in SODA are
more consistent, specific, and (surprisingly) natural than those in prior
human-authored datasets.
Using SODA, we train COSMO: a generalizable conversation model that is
significantly more natural and consistent on unseen datasets than
best-performing conversation models (e.g., GODEL, BlenderBot-1, Koala, Vicuna).
Experiments reveal COSMO is sometimes even preferred to the original
human-written gold responses. Additionally, our results shed light on the
distinction between knowledge-enriched conversations and natural social
chitchats. We plan to make our data, model, and code public.Comment: EMNLP 2023. Dataset, model, and code can be found at
https://hyunw.kim/sodavers
Value Kaleidoscope: Engaging AI with Pluralistic Human Values, Rights, and Duties
Human values are crucial to human decision-making. Value pluralism is the
view that multiple correct values may be held in tension with one another
(e.g., when considering lying to a friend to protect their feelings, how does
one balance honesty with friendship?). As statistical learners, AI systems fit
to averages by default, washing out these potentially irreducible value
conflicts. To improve AI systems to better reflect value pluralism, the
first-order challenge is to explore the extent to which AI systems can model
pluralistic human values, rights, and duties as well as their interaction.
We introduce ValuePrism, a large-scale dataset of 218k values, rights, and
duties connected to 31k human-written situations. ValuePrism's contextualized
values are generated by GPT-4 and deemed high-quality by human annotators 91%
of the time. We conduct a large-scale study with annotators across diverse
social and demographic backgrounds to try to understand whose values are
represented.
With ValuePrism, we build Kaleido, an open, light-weight, and structured
language-based multi-task model that generates, explains, and assesses the
relevance and valence (i.e., support or oppose) of human values, rights, and
duties within a specific context. Humans prefer the sets of values output by
our system over the teacher GPT-4, finding them more accurate and with broader
coverage. In addition, we demonstrate that Kaleido can help explain variability
in human decision-making by outputting contrasting values. Finally, we show
that Kaleido's representations transfer to other philosophical frameworks and
datasets, confirming the benefit of an explicit, modular, and interpretable
approach to value pluralism. We hope that our work will serve as a step to
making more explicit the implicit values behind human decision-making and to
steering AI systems to make decisions that are more in accordance with them
Faith and Fate: Limits of Transformers on Compositionality
Transformer large language models (LLMs) have sparked admiration for their
exceptional performance on tasks that demand intricate multi-step reasoning.
Yet, these models simultaneously show failures on surprisingly trivial
problems. This begs the question: Are these errors incidental, or do they
signal more substantial limitations? In an attempt to demystify Transformers,
we investigate the limits of these models across three representative
compositional tasks -- multi-digit multiplication, logic grid puzzles, and a
classic dynamic programming problem. These tasks require breaking problems down
into sub-steps and synthesizing these steps into a precise answer. We formulate
compositional tasks as computation graphs to systematically quantify the level
of complexity, and break down reasoning steps into intermediate sub-procedures.
Our empirical findings suggest that Transformers solve compositional tasks by
reducing multi-step compositional reasoning into linearized subgraph matching,
without necessarily developing systematic problem-solving skills. To round off
our empirical study, we provide theoretical arguments on abstract multi-step
reasoning problems that highlight how Transformers' performance will rapidly
decay with increased task complexity.Comment: 10 pages + appendix (21 pages
- …